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Why OPeNDAP?? 


¢« Check metadata remotely (in various forms) 

¢ Obtain the subset of data easily and efficiently 

¢ Hide the original data sources 
¢ Hierarchical Data Format (HDF) versions 4 and 5 
¢ Network Common Data Form (NetCDF) 
¢ Geospatial Tagged Image File Format (GeoTiFF) 

¢ Many popular earth science tools can visualize and 
analyze the data via OPeNDAP 

¢ OPeNDAP output (including subsets) can be downloaded 
as other formats 


1 Open-source Project for a Network Data Access Protocol 2 


HDF(5) Hyrax modules 


¢ The visualization of HDF(5) data via Hyrax 
¢ What NASA users request the most 


Visualize A Soil Moisture Active Passive (SMAP) HDF5 variable via Hyrax 


Representative soil moisture measurement for the Earth based grid cell. 
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HDF5 handler and NcML' 


NcML module can be used with HDF handlers to provide the 
missing Climate and Forecast Metadata (CF) conventions 
information 

i CAUsers' \Des P-meetin g-slides\text\tes neml - Notepad++ e_Xx 


File Edit Search View Encoding Language Settings Macro Run Plugins Window ? 
 oeHRE SFB 4 OH Ac|*el_es(BQl>i1BS SHdaBlary 
‘G testnem! | 
| 1 <?xml version="1.0" encoding="UTF-8"?> 
2 <netcdf location="/data/hdf4/2006001-2006005.s0454pfrt-—bsst.hdf" xmilns="http://www.ur 
| 


<!-- Although variable@type attribute is technically required by the schema, i have i 


using type=""_to mean “whatever type a variable with the given name has" --> 
<variable name="bsst" type=""> 


<!-- Test adding new attributes to the variable --> | 


<attribute type="string" name="units" value="Celsius"/> 
<attribute name="add offset" orgName="add off" type="Float32"/ > 


</variable> 


length: 602 lines: 17 


1 NetCDF Markup Language 


HDF5 handler and file NetCDF 


¢ File NetCDF module can work with HDF 
handlers to convert HDF files to NetCDF-3 
or NetCDF-4 classic files that follow the 
CF conventions. 
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_HDEFS5 handler and file NetCDF 


OPeNDAP Server Dataset Query Form - Mozilla Firefox = 
File Edit View History Bookmarks Tools Help 
[2] OPeNDAP Server Dataset Query... | <P | 


- localhost:8080/opendap/data/hdf4/2006001-2006005.s0454pfrt-bsst.hdfua) Fav Google mm) Jb @ 


OPeNDAP Server Dataset Access Form 


Action: | Get ascii |{ Get as NetCDF 3 || Get as NetCDF 4 || Binary (DAP) Object || Shc 


Data URL: ttp://localhost:8080/opendap/data/hdf4/2006001-2006005.s0454pfrt-bsst.hdf | 


Global] HDF_GLOBAL.dsp SubImageId: 0 
. HDF GLOBAL.dsp SubImageName: BSST E 
Attributes: |ipF GLoBaL.dsp ModificationDate: 20070409 

HDF GLOBAL.dsp ModificationTime: 120728 


HDF GLOBAL.dsp SubImageFlag: 64 iy 
HUHNE CIARAIL dAcn CithTmanaTitila: \\\\MIC Dactana Ctamn 


Variables: © bsst: Array of 16 bit Unsigned integers [lat = 0..4095][lon = 0..8 
, ci 
E D) 
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Use NcML and file NetCDF to work with HDF5 handler 


bsst bsst 


ae | setnaad_gee== 


bsst (Temp) bsst (Celsius) 
0.0 7:0 139 26.9 27.8 34.8 “ 4.0 109 17.9 24.8 31.8 
AVHRR’ via Hyrax directly The NetCDF file of AVHRR’ 


via Hyrax file NetCDF module 


' Advanced Very High Resolution Radiometer 8 
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service Chain to access HDF5 via Hyrax 


OPeNDAP clients || ( ae See 
(netCDF, Ferret, .| 
re) 


Panoply, . 


Main engine 
(===) HDF(5) handlers 


Alternative output 
File NetCDF Hyrax core 


Supplement Info 


NcML 
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DAP'4 support in the HDF5 handler 


¢ CF option 
— DAP4 strictly mapped from DAP2 
— Dataset Metadata Response (DMR) replaces Dataset 
Descriptor Structure (DDS) and Dataset Attribute Structure 
(DAS) 
¢ Default option 
— HDF5 group to DAP4 group 
— HDF5 signed 8-bit and 64-bit integers to DAP4 
— HDF5 dimensions following the NetCDF-4 to DAP4 mapping 
¢ Fully support NetCDF-4 (both classic and enhanced) 
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Interoperability ennancement 


¢ CF option with DAP2 


— Provide a way for service providers to check if 
there are any objects ignored when mapping 
from HDF5 to DAP2 


¢ An example: DAP2 doesn't support 64-bit integer, 
HDF5 supports 
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Performance Improvement 


¢ Reducing DDS and DAS access time 
— Caching DDS and DAS in memory 
— Caching DAS in the disk 
¢ Reducing data access time 
— Caching the raw in the disk 
— Best if the data is compressed 
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Other New Features 
¢ Support the access of HDF-EOS5 


sinusoidal projection in the HDFS 
OPeNDAP handler 
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ACCESS HDF5 via Hyrax in Cloud 


¢ Three architectures 


¢ HDF5 handler can be enhanced for the 
future work of Architectures 2 and 3 
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Archit. #2: Files With HTTP’ Range-Gets 
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Request 


Response —> 


Index 
Collections/VirtualOrg 


Dataset: 

- $3-ID 

- DAP-ID 

- Metadata 

- Per Variable Range Access 
Info 


Range End 


* Index Location TBD 
* Read metadata (DMR/DDS/DAS, coordinates?) from index. 
* Read data from S3 object using HTTP range GET. 
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Archit. #3: HDF5 Datasets as S3' Objects 
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Index 
Collections/VirtualOrg 


Dataset: 

- $3-ID 

- DAP-ID 

- Metadata 

- Per Variable Shard ID/ 
Access Info 


Retrieval 


* Index Location TBD 
* Read metadata (DMR/DDS/DAS, coordinates?) from index. 
* Read data shard from S3 using simple GET. 
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What can be improved’? 


¢ Current 


— A2: Range-Gets index per chunk 


— A3: make an HDF5 chunk in a variable as an 
S53 object 


¢ Future 


— Add an option to the HDF5 handler 
¢ Range-Gets index per variable 
¢ Make an HDF5 variable as an S3 object 
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